THE QUALIIY 
OF S0F1WARE 
IS MANAGEABLE 

How do you measure and ensure 
the quality in software? Here's what 
it means for manufacturing. 



BY LAWRENCE GOULD 

Software can never be totally 
bug-free. It is commonly accept- 
ed that no software with more 
than 10,000 lines of code has 
ever formally been proved correct. 
Given this, one need not look at the 
government's Star Wars project to 
shudder. Bugs in manufacturing plan- 



ning and control software can prove 
devastating in actual costs, lost pro- 
duction, and lost customers. 

A software vendor's quality as- 
surance can eliminate most of the kill- 
ing bugs so that the user can be rea- 
sonably sure that the program will 
function as specified. However, users 
typically expect the quality of soft- 
ware to be better than that of any 




other product they buy. They 
shouldn't, though. 

"You have to treat software like 
any other manufactured product," 
says Lou Mazzucchelli, chief techni- 
cal officer of Cadre Technologies Inc. 
"All the same rigor should apply. 
There's not a company in the world 
that doesn't have a well-formulated 
set of procedures and control mecha- 
nisms for either manufacturing or 
changing their product. It is virtually 
the same whether for metalworking 
or for software manufacturing. " 

Adds John Perkins, software 
programmer for Dynamics Research 
Corp., "The question shouldn't be 
whether the software is right or 
wrong. It should be whether the soft- 
ware is useful or not useful. And 
when it breaks, do you know it?" 

Software quality is often ex- 
pressed in vague terms such as reli- 
ability, maintainability, and efficien- 
cy — terms with different meanings 
depending on who is talking, the user 
or the programmer. As a result, a va- 
riety of software users, industry 
groups, and standards organizations 
are working to define those terms in 
precise, measurable parameters. 

For instance, the Rome Air De- 
velopment Center at the Griffiss Air 
Force Base has spent 10 years devel- 
oping a methodology for specifying 
and evaluating software quality attri- 
butes. Both the Institute of Electrical 
and Electronics Engineers and the 
American Society for Quality Control 
have committees that develop stan- 
dards on such topics ranging from 
software development to software 
quality. Some of these quality attri- 
butes are functionality, performance 
and efficiency, code coverage, accu- 
racy, configuration and portability, 
maintainability, redundancy, and con- 
formance to standard. 

Some quality attributes are easily 
quantifiable. With performance, ei- 
ther the software does what it is sup- 
posed to do in a given amount of time 
or it doesn't. With adherence to stan- 
dards, either the software meets a 
list of functions or it doesn't. 

However, the metrics for some 
quality attributes are harder to de- 
fine. One such metric is the McCabe 
Complexity Measure, which counts 
such code features as the number of 
branches in the software program. 
This measure assumes that with 
more branches or paths through a 
program, the more complex it is, and 
the more likely it is to have an error. 

The most important software 
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quality metric, says Marty Browne, 
vice president of applications devel- 
opment at ASK Computer Systems 
Inc., "is whether that software per- 
forms the way the specification and 
the documentation say it is supposed 
to perform. " 

Adds Nick Stewart, software 
quality engineer at Rockwell Interna- 
tional Satellite & Space Electronics 
Division and chairman of the Soft- 
ware Quality Technical Committee 
for the ASQC, "The definition of 
quality software is like any other 
product in the world: It's in the eyes 
of the beholder." 

Not surprisingly, quality costs, 
or rather, non-conformance to 
software requirements costs, 
go up as the life cycle pro- 
gresses. According to Stewart, 70% 
of the life-cycle cost of software is in 
the maintenance phase — that is, 
when the software is in the custom- 
er's hands. Moreover, 60% of all 
software errors can be traced to the 
requirements at the design phase. 

Explains Mazzucchelli, "Pro- 
grammers generally do not make a 
mistake translating their intent into 
code, and syntactic errors are easy 
to find. However, if the programmers 
are working from bad specifications, 
they'll do a good job of translating the 
bad specifications into code. " 

Continues Stewart, "It costs as 
much as six to ten times to correct an 
error discovered during the mainte- 
nance phase as it would if the error 
were caught in the design phase. In 
some highly complex aerospace appli- 
cations, that estimate goes as high as 
one hundred times." 

According to studies by AT&T 
Bell Labs, it costs nothing to fix an 
error in the initial design phase, $100 
to fix the same error once it is coded, 
and $10,000 to fix the problem afi;er 
the software is released. 

Stewart explains: "The cost of an 
error as the software product passes 
into the requirements phase of devel- 
opment would include the time spent 
by those who negotiated the change; 
namely, the contracting department 
and the analyst who performed the 
study and made changes to the re- 
quirements documents, plus printing. 
These costs shouldn't be high. 

"At the design phase, all out- 
standing issues of requirements are 
supposed to be resolved. Any 
changes from this point on will incur 
the additional costs of modifying a 
solid set of documents. 



• Requirements incorrect or 
Misinterpreted 

• Functional Specification Incorrect or 
Misinterpreted 
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• Design Error Involving Several 
Components 

• Error in Design or Implementation of 
Single Component 
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Error Due to Previous Miscorrection 
of an Error 
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"The cost of an error in the cod- 
ing phase includes the cost of all 
those who participated in the analysis 
to resolve the error, and the time to 
implement the change. This cost is 
compounded by the number of inter- 
faces to other modules. If an error is 
not a simple correction to logic or 
naming convention, the time and la- 
bor to correct it can be extensive. It 
is not uncommon to find 40% of the 
development cost in the test phase. 

Says Ted Williams, president of 




Comae Systems Corp. , a producer of 
computer-aided maintenance man- 
agement software, "Quality software 
is money in the bank. It just makes 
good sense from a business stand- 
point to have the highest-quality soft- 
ware application possible. Whatever 
we can do to improve that quality 
during the process of developing, re- 
leasing, and upgrading that software, 
the less time we have to spend fixing 
the software, repairing errors, and 
potentially damaging our image." 



Whof s Bugging You? 

Software errors, called bugs, come in several varieties: 

Typographical and coding errors are typically transcription errors that are easily 
found through proofreading and automated debuggers that check each line of code 
for operability. A keystroke error, though, may appear as a logical error. For example, 
in C language, a double equal ( = = ) sign is a test between the two objects on either 
side of the equal signs, but a single equal ( = ) sign assigns the value to right of the 
sign to the object on the left. 

Logic errors can be as simple as a mindless mistake, such as not writing what 
was intended or using one identifier for another, or they can be the result of a pro- 
grammer losing the thread of a complex analysis translated into code. 

Project management errors occur because one programmer assumed that an- 
other programmer was handling an aspect of the software project. These errors in- 
clude missing code critical for the software to operate, missing functionality, multiple 
labels for the same data elements, and unlinkable software modules. 

Interface errors occur when two or more software modules within the same 
program cannot exchange data because data elements are not defined properly, are 
mislabeled, or are in some other way insufficient. Interface errors are often the result 
of inadequate project management, but they can also be design errors, especially in 
revised or added modules to an existing software program. 

Misinterpretation of specifications are not necessarily errors in software code. 
Explains Max Hitchens of HEI Corp., "In this case, the software does what I tell it to 
do, not what I want it to do." 

"Does the program perform the way it was designed to perform or perform in a 
way that solves the customer's problem?" says Marty Browne of ASK Computer 
Systems. "Unfortunately, these may be two different issues." 

Explains Fabrizio, "System malfunctions resulting from misinterpreted specifica- 
tions are perceived as software errors by the customer, but the system may be doing 
exactly what the developer had intended! In this case, the developer and the custom- 
er have interpreted the specification differently." 
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To combat costs, software must 
be developed and tested according to 
rigorous standards. Software devel- 
opment typically consists of seven 
phases that generally follows design, 
code, and test: 

Conceptualization: the custom- 
er's initial description of the applica- 
tion, including needs, hardware, in- 
terrelated automated systems 
(hardware and software), user exper- 
tise, and future requirements. 

Functional requirements: a more 
detailed description of the software, 
including functionality, performance 
requirements, and the user or ma- 
chine interface. 

Functional design: a detailed de- 
sign of the software created by the 
software vendor, including software 
architecture, modules, interfaces, 
and data requirements (inputs, out- 
puts, and how they are processed). 

Implementation: writing the soft- 
ware code and initial debugging. 



Testing: checking out the soft- 
ware for logic, functionality, inconsis- 
tencies, unresolved linking, and other 
errors through a variety of testers, 
including static testers, regression 
testing, simulation, and emulation. 

Installation and checkout: a full- 
functional test of the software in its 
operational environment. 

Maintenance: modifications to 
the released software to correct soft- 
ware errors, revise or add functional- 
ity, or port to other systems. 

Software has to steep like good 
wine," explains WilHams. "It 
takes a number of iterations to 
remove software inconsistencies 
and bugs. And there's no shortcut to 
doing that; software testing takes 
time and effort. " 

Robert Fabrizio, director of fac- 
tory systems, ITP boston Inc., 
agrees. "There seems to be a strong 
indication that the longer you spend 



in design, the less time is required in 
coding and debugging a piece of soft- 
ware. It's a trade-off between design 
time and debugging." 

However, there is a catch to 
testing software: How do you know 
you're done? "There is no good an- 
swer to that question, " replies Fabri- 
zio. "As a system becomes more 
complex, it's generally understood 
that you can never test every possi- 
ble condition that could ever arise. 
However, a good measure is the time 
between errors. The longer you run 
the system to find the next error, the 
closer you are to the end of the test. 
In fact, some users will specify that 
acceptance tests run 40 hours, 10 
days, or whatever, without an unre- 
coverable error." 

Although no test can replace a 
user in an actual operating environ- 
ment, software is typically used to 
test and ensure the quality of soft- 
ware. These software tools are avail- 
able at both the front and back ends 
of software development. For in- 
stance at the front end, computer- 
aided software engineering (CASE) 
provides software design with what 
computer-aided engineering provides 
hardware design. 

CASE typically allows program- 
mers to write, compile, test, and de- 
bug code at the same terminal. It at- 
tempts to put as much of the 
development effort up front in the 
software design cycle — namely, dur- 
ing the functional requirements and 
design phases — thereby reducing the 
errors (and costs) that result from 
faulty detailed design specification. 

One CASE product — Teamwork 
from Cadre Technologies — provides 
real-time modeling, systems analysis, 
system simulation, source code gen- 
eration, language tools, debuggers, 
software performance analysis, and 
verification. Teamwork allows pro- 
grammers to hypothesize software 
designs and simulate performance be- 
fore investing time in development. 

Other Teamwork modules, from 
Cadre's MicroCASE Division, pro- 
vide back-end support by emulating 
off-the-shelf microprocessors and 
software analyzers, thereby helping 
programmers observe code execu- 
tion and performance. 

Another approach to back-end 
performance testing is to simulate or 
emulate the manufacturing equipment 
and then run the target software as if 
it was in an actual operation. Explains 
Max Hitchens, president of HEI 
Corp., a supplier of real-time emula- 



How Does Software Measure Up? 

Some of the quality attributes of software are as follows: 

Functionality measures whether the software does what it is supposed to do. 
Functionality must be measured dynamically: When the software is being used and 
being compared to the initial customer specifications. 

Performance and efficiency measures whether the software is fast enough com- 
pared to an existing software product or previous revision, or perceived as fast by the 
user. For example: Does the software keep up with the hardware — that is does the 
disk access speed slow the software? Does it keep up with the operator — that is, 
does the cursor move with the mouse controller? Does it keep up with the applica- 
tion — that is, can the software collect the data when needed? 

Code coverage measures whether all of the code in a program is being used. 
Excess code can affect a program's performance, efficiency, and reliability. 

Accuracy measures whether the software counted the data correctly and wheth- 
er it evaluated an action, especially an emergency action, properly. To a large extent, 
a test of the software's accuracy is really a test of its functionality. 

Configuration measures whether the software operates as required on the plat- 
form specified, including hardware and its operating system, display monitor, periph- 
erals, networking system, and other applications software. Equally important, does all 
the software functionality work on every configuration of that hardware; for example, 
does the software display data on a monochrome as well as on a color screen? 

Portability is another aspect to the configuration attribute; that is, can the soft- 
ware be used on different hardware platforms. For example, does a software applica- 
tion written in MS-DOS work for all PC-compatibles? 

Maintainability measures the ease that software can be changed, enhanced, 
fixed, and updated. 

Redundancy, also known as fault tolerance, resource recoverability, or error 
recovery, is probably one of the most critical aspects of software. This test evaluates 
whether the software handles spurious inputs and errors 'gracefully." For example, if 
the operator keys in a six-digit number when the software expects a five-digit num- 
ber, what should the software do? 

Conformance to standards determines whether the software — either its coding 
structure or its functionality — adheres to a standard, if one exists. In some applica- 
tions, translating a standard into software is critical. For example, aerospace and 
defense industries must meet stringent Department of Defense quality and traceabili- 
ty specifications when assembling defense systems. In accounting, there are "ac- 
cepted accounting practices," which directly affect the profit and loss of a company. 

In manufacturing, explains Christopher Gray, president of Gray Research, "a 
measure of the quality of manufacturing resource planning (MRP II) software is how 
well does it conform to the functions outlined in the Standard System, first offered in 
1976 by Oliver Wight. Specifically, does the MRP II software include all the function- 
ality for master production scheduling, demand management, time-phased require- 
ments planning, sales and operations planning, and so on?" 
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tion systems, "A PC-based simulator 
can be connected to programmable 
controllers or computers to both test 
the software implementation and de- 
bug the control system. 

"As an emulator, it can replace 
conveyors, storage systems, guided 
vehicles, robots, and other physical 
computer-based equipment so that 
the control system can be tested in 
real time. The emulation can show 
that the software is operational and 
that it meets customer functional and 
performance specifications before in- 
stallation, eliminating expensive and 
time-consuming field testing. " 

Unlike simulation and emulation, 
which tests software dynamically, 
other back-end software tools pro- 
vide static tests for software quality. 
Adamat from Dynamics Research de- 
tails whether Ada language source 
code reflects Department of Defense 
software engineering goals. The soft- 
ware test system checks more than 
150 Ada-specific metrics to deter- 
mine if the code adheres to the ac- 
cepted Ada quality goals of maintain- 
ability, reliability, and portability. 

A similar product. Scoreboard, 
from TravTech Inc., performs static 
code quality analysis for COBOL pro- 
grams. This package rates the com- 
plexity and structure of software 
based on standards from independent 
IEEE studies and weighted by the 
user. The program outputs an overall 
score from 0 to 100 for the software 
package, as well as scores individual 
software metrics so that the user can 
determine which aspects of the soft- 
ware need to be revised. 

Software is purchased caveat 
emptor, let the buyer beware. Be- 
cause of that, software vendors have 
been accused of "getting away with 
murder." Some, of course, are re- 
versing the status quo. Says Comae's 
Williams, "If somebody does some- 
thing totally unusual and tells us, we 
either fix the bug in the next release 
or give the user a way to work 
around the bug, such as an instruc- 
tion or a quick fix sent by modem. " 

Sun Microsystems Inc. (Moun- 
tain View, Calif) uses its Network 
Software Environment to manage all 
the objects created while developing 
software, including data, modules, 
and documentation. Explains George 
Symons, CASE product line manager 
at Sun, "With operating system soft- 
ware, there is the issue of releasing 
and re-releasing software and manag- 
ing multiple versions of that software 
for different architectures. To do 



that, we must have a release mecha- 
nism to ensure the distribution of the 
right pieces of software and docu- 
mentation are current. " 

But how can the software cus- 
tomer ensure the quality of software? 
"The first thing you can do to im- 
prove the quality of software," says 
Mazzucchelli, "is to get it right from 
day one, and that means you have to 
pay a lot of attention to the specifica- 
tion and detailed design of the soft- 
ware, which points back to knowing 
your business and what you want. " 

Explains Fabrizio, "Software de- 
velopment ideally should be a joint ef- 
fort between the customer and the 
software vendor. The customer 
should be involved up front in the 
functional specification of the system. 
The customer should also be involved 
in the back end by defining the accep- 
tance test plan, to verify whether the 
software meets specifications and the 
customer's needs, and run the test to 
ensure proper operation, fault dispo- 
sition, and acceptance." 

The customer should investigate 
the software itself its age, the fre- 
quency of software updates, and the 
vendor's future software plans. Says 
Mazzucchelli, "The customer should 
not necessarily care about the soft- 
ware language or how the program is 
written except as it applies to longev- 
ity. For example, if the program is 
written in a proprietary language for 
one brand of computer, be skeptical. 
You might want to move that soft- 
ware system in the future to another 
kind of computer. " 

Most important, open communi- 



cation, typically called customer sup- 
port, between the software vendor 
and user must exist. This communi- 
cation runs both ways: Not only 
should users be able tell vendors of 
bugs, but if the vendor finds or hears 
of a bug, some mechanism should ex- 
ist for vendors to tell users about that 
bug and when it will be corrected. 
For instance, ASK has a users con- 
ference and customer surveys, and 
updates its customers about bugs. 

However, as with purchasing any 
other product or service, the user 
should investigate the vendor's repu- 
tation. The customer should visit 
customer references; see what their 
application is, how the software is 
used, and its benefits. Adds Williams, 
"Get an objective assessment as to 
how well the product is holding up 
and how well the vendor supports it. 
Then deal with the vendor. " AM 
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